Skip to content

Moonshine speech engine + Wayland overlay fixes (v0.3.0)#61

Merged
JRufer merged 9 commits into
masterfrom
claude/moonlight-implementation-review-mqcjj9
Jul 5, 2026
Merged

Moonshine speech engine + Wayland overlay fixes (v0.3.0)#61
JRufer merged 9 commits into
masterfrom
claude/moonlight-implementation-review-mqcjj9

Conversation

@JRufer

@JRufer JRufer commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Summary

Two features plus the 0.3.0 version bump.

1. Real Moonshine ONNX speech-to-text backend

The moonshine backend was selectable in Settings/config/docs but had no implementation — it silently ran whisper-cpp, and the "model not downloaded" guards were skipped for it, so a Moonshine selection could leave a permanently silent, unexplained pipeline.

Implemented for real against the current upstream ONNX export (two graphs: encoder_model + a KV-cached decoder_model_merged with use_cache_branch), greedy decode from SOT→EOT, tokenizer embedded in the binary. Model files download on demand to ~/.local/share/voxctrl/models/moonshine/<size>/. Wired a download button + "not-in-this-build" notice into Settings → Engine, and fixed the two model-check guards to only skip the Whisper check when Moonshine is actually compiled in.

Moonshine is now built into every release (--features moonshine in build_appimage.sh and release.yml); ONNX Runtime links statically, so binaries stay self-contained.

2. Overlay always-on-top + positioning on Wayland

On native Wayland a regular client can't control its own stacking or position, so the overlay fell behind windows and was mis-placed (scale-offset from center). Fixed by running the overlay through XWayland (where KWin honors _NET_WM_STATE_ABOVE + positioning) and computing the position inside the overlay from its own monitor scale (the app now sends the anchor, not pre-scaled pixels). Confirmed working on KDE Plasma Wayland.

3. Cleanup

Deduplicated expand_tilde/model-dir helpers into a shared util module; optimized the Moonshine decode loop (precomputed input binding, vector KV caches, no per-step allocations/string-matching); removed the now-orphaned app-side overlay coordinate math.

Testing

  • cargo test -p voxctrl-inference — 34 pass (default), 48 with --features moonshine.
  • cargo check -p voxctrl-app clean; overlay anchor_y unit tests pass.
  • Frontend: npm run build, npm run test:unit (25), svelte-check (0 errors).
  • Runtime-verified by the maintainer: Moonshine transcription works; overlay stays on top at the configured position on KDE Wayland.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6

claude added 9 commits July 5, 2026 02:13
The "moonshine" backend was selectable in Settings, config, and docs but had
no implementation: choosing it silently substituted whisper-cpp (a warning-only
log), and the startup/keypress "Whisper model not downloaded" guards were
skipped for a Moonshine selection — so a user who picked Moonshine could end up
with a permanently silent dictation pipeline and no indication why.

Backend (crates/voxctrl-inference/src/moonshine.rs, feature-gated):
- Runs the four upstream Moonshine ONNX graphs (preprocess, encode,
  uncached_decode, cached_decode) via ort, with greedy autoregressive decoding
  from the start-of-transcript token to the end-of-transcript token, capped by
  audio duration.
- Decodes token ids to text with the model's tokenizer.json (tokenizers crate).
- Binds tensors to graph inputs positionally and detects the optional seq_len
  input and KV-cache arity from the graph, so it tolerates export revisions.
- Downloads the ONNX graphs + tokenizer on demand into
  ~/.local/share/voxctrl/models/moonshine/<size>/, or reads a manual copy.

Wiring and correctness:
- build_backend constructs the real backend when compiled; otherwise it keeps a
  clear whisper-cpp fallback.
- MOONSHINE_COMPILED exposes whether the backend is built in; the two src-tauri
  model-download guards now only skip the Whisper-model check when Moonshine is
  actually compiled (a non-Moonshine build still needs the Whisper model).
- New tauri commands (moonshine_available, check_moonshine_downloaded,
  download_moonshine_model) drive a Moonshine download button + status and a
  "not included in this build" notice in Settings → Engine.

Moonshine is an opt-in compile feature (--features moonshine), like cuda/vulkan,
since it links ONNX Runtime. Docs updated accordingly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
Make the Moonshine engine ship in every distributed artifact so users can pick
either speech engine, the same way whisper-cpp is always available:

- build_appimage.sh: add `moonshine` to both the CUDA and CPU `tauri build`
  invocations.
- release.yml: add `moonshine` to every artifact build — Linux CPU AppImage/deb,
  Linux Vulkan AppImage/deb, Linux CUDA deb, and Windows CPU/CUDA installers.

ort's `download-binaries` (default) links ONNX Runtime *statically* into the
executable, so these binaries stay self-contained — no extra .so/.dll to bundle,
and the AppImage/NSIS packaging is unchanged. The trade-off is that these build
steps now fetch the ONNX Runtime static lib at compile time (needs network) and
produce a somewhat larger binary.

The `moonshine` cargo feature is kept as the toggle (so a plain
`cargo check --workspace` stays fast and network-free); it is simply enabled by
every real build/release command.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
The download failed because the backend targeted an outdated 4-file model layout
(preprocess/encode/uncached_decode/cached_decode) that no longer exists upstream.
The current Moonshine ONNX export (matching useful-moonshine-onnx) is two graphs:

  onnx/merged/<size>/float/encoder_model.onnx
  onnx/merged/<size>/float/decoder_model_merged.onnx

Rewrite the backend to that architecture:
- Encoder: raw 16 kHz audio -> hidden states (with optional attention_mask).
- Merged KV-cached decoder driven by a `use_cache_branch` toggle and named
  `past_key_values.<layer>.<decoder|encoder>.<key|value>` inputs. Greedy decode
  from SOT (1) to EOT (2): the first step feeds empty caches; later steps reuse
  the decoder self-attention cache while the encoder cross-attention cache stays
  frozen. Decoder inputs are bound by name (they include optional masks and many
  cache tensors); empty rank-4 caches are built via the allocator since
  `from_array` rejects their 0-sized dimension.
- Per-size geometry: tiny = 6 layers, base = 8 layers (8 kv-heads; head_dim
  36/52), validated against the loaded graph.

The tokenizer is bundled (embedded from assets/moonshine_tokenizer.json) rather
than downloaded, matching upstream, so model download now fetches just the two
ONNX graphs. Docs updated to describe the 2-graph layout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
Reduce duplication and speed up the decode hot path.

Deduplication:
- Extract the twice-copied `expand_tilde` and the models-base-dir construction
  into a shared `util` module; whisper-cpp and Moonshine now both use it. Move
  the tilde tests alongside the helper.

Decode-loop optimization:
- Resolve each decoder input to its source once at load time into a
  `decoder_plan`, then bind inputs positionally at run time. This drops the
  per-step HashMap rebuilds, per-input string matching, and name-string clones
  that ran on every one of up to 192 decode steps.
- Store KV caches in a Vec (emit order) instead of a name-keyed HashMap, and
  skip re-copying the encoder cross-attention caches on cached steps (they are
  fixed after the first step) — only the decoder self-attention caches are read
  back and grown.
- Allocate the all-ones attention mask only when a graph actually consumes it.

No behavior change; 48 tests pass with the feature enabled, 34 without.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
The overlay's always-on-top level was applied only once, when the window was
first mapped, and never re-asserted. On X11 a window is free to lose its topmost
position afterwards — another window taking `_NET_WM_STATE_ABOVE`, a fullscreen
app, or the compositor re-stacking between dictations — and the overlay would
stay behind, sometimes fully hidden, for the rest of the session.

Re-assert the always-on-top level (and click-through) via a new
`reassert_topmost` helper:
- when the overlay activates for a new dictation (idle → visible), so it comes
  back to the front even if it fell behind while idle;
- on a ~1s heartbeat while visible, so it can't linger behind a window that
  came forward mid-dictation.

The helper toggles Normal → AlwaysOnTop rather than re-setting the same level,
because re-adding an already-set state is a no-op that won't re-raise; the
toggle forces the WM to re-evaluate stacking. Both calls happen within one
render tick with no frame presented between them, so there is no visible flicker.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
The always-on-top re-assertion toggled the window level (Normal → AlwaysOnTop)
every time it was applied, including on the very first map. Toggling the state
while the surface is still being mapped left some compositors with the window
unpresented until the next state change — so at launch no overlay appeared until
re-selecting the overlay style forced another update.

Only toggle when re-raising an already-mapped window (activation after idle, and
the visible heartbeat). On the first map, set AlwaysOnTop directly, as before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
Re-asserting the always-on-top level disturbs the window's position: winit's
set_window_level re-syncs the window geometry from winit's own position
tracking (and some WMs re-place a window on a state change), moving the overlay
toward the WM's default spot. The render tick then failed to correct it because
Slint's set_position is cached — re-sending the same (unchanged) value is a
no-op, so no corrective request reached the compositor and the overlay stuck
near center instead of the configured position.

For a short window (~100 ms) after every level change — the first map, each
activation, and each heartbeat — push the position through winit's
set_outer_position directly, which is not cached and re-asserts the geometry.
Steady-state frames keep using Slint's cheap cached set_position.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
On a native Wayland session a regular client cannot control its own stacking or
position — the compositor decides both — so the overlay's always-on-top and
configured placement were silently ignored, leaving it able to fall behind other
windows and mis-placed (offset from center by the display's scale factor).

Two changes:

1. Run the overlay through XWayland on Wayland sessions (spawn with
   WAYLAND_DISPLAY removed when an X server is reachable). As an X11 client,
   KWin honors _NET_WM_STATE_ABOVE and window positioning. On a pure X11 session
   this is a no-op.

2. Compute the overlay position inside the overlay process from its own winit
   monitor + window size, instead of receiving pre-scaled pixel coordinates from
   the app process. The two processes can see different display scales, which
   put the window at a fraction of the intended offset (up-and-left on HiDPI).
   The position messages now carry the anchor ("top"/"bottom"/"center") + monitor
   preference; the overlay resolves pixels in a single coordinate space and
   applies them via winit (uncached) so a level change can't strand it.

Removes the now-unused app-side coordinate math (calculate_overlay_coordinates /
calculate_overlay_y); the vertical-anchor formula moves to the overlay as a pure,
tested `anchor_y` helper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
Moonshine ONNX speech-to-text backend (opt-in, now built into releases) and the
Wayland overlay fixes (always-on-top + correct positioning via XWayland).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
@JRufer JRufer changed the title Implement real Moonshine ONNX speech-to-text backend Moonshine speech engine + Wayland overlay fixes (v0.3.0) Jul 5, 2026
@JRufer JRufer marked this pull request as ready for review July 5, 2026 19:34
@JRufer JRufer merged commit 109d48a into master Jul 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants